The present invention relates to training of cognitive computing systems, and more specifically, to techniques and mechanisms for improving the results generated by a Question and Answer system by analyzing and ranking ground truth provided by subject matter experts and feeding back ranking and evaluation information of the ground truth to the subject matter experts.
With the increased usage of computing networks, such as the Internet, users can easily be overwhelmed with the amount of information available from various structured and unstructured sources. However, information gaps abound as users try to piece together what they believe to be relevant during searches for information on various subjects. To assist with such searches, research has been directed to creating cognitive systems such as Question and Answer (QA) systems that take an input question, analyze the question, and return results indicative of the most probable answer or answers to the input question. QA systems provide automated mechanisms for searching through large sets of sources of content, e.g., electronic documents, and analyze them with regard to an input question to determine an answer to the question and a confidence measure as to how accurate an answer to the question might be.
The IBM Watson™ system available from International Business Machines (IBM) Corporation of Armonk, N.Y. offers several services that can be used to build such QA systems. The IBM Watson™ system is an application of advanced natural language processing, information retrieval, knowledge representation and reasoning, and machine learning technologies to the field of open domain question answering.
The quality of the responses provided by a QA system is tied to the training provided to the system. When a cognitive system is trained, ground truth is provided to the system. The quality of system training, and in turn, the quality of the cognitive system is determined by the quality of the ground truth used to train the system. The primary feedback for a set of ground truth is either a manual inspection by experts at training cognitive systems, or an evaluation of the cognitive system after a lengthy system training phase. Both of these solutions yield long feedback cycles and make it difficult for users to quickly get a sense of the quality of the ground truth they are providing to the system.
Various publications describe training a cognitive system, using subject matter experts or other methods to provide ground truth to a system. System and Method for Generating Question Type Distribution of a Training Data Set in a Question/Answering System, IPCOM000239801D (Dec. 2, 2014), teaches that “training data may be manually created by subject matter experts.”
Drozda et al., Online Crowdsource System Supporting Ground Truth Datasets Creation, describes a “system for creating ranked image datasets based on user feedback.”
Improving User Feedback In A Question Answering System For Indirect Answers, IPCOM000239021D (Oct. 1, 2014) teaches that “Before training, human experts gather a set of sample questions and on-topic answers to those questions. Some of those answers will be correct, while others will be incorrect but still on-topic. During training, the system generates candidate answers and assigns feature values to those candidate answers. An answer-is-on-topic model is built that assigns high scores to known on-topic candidate answers and low scores to other candidate answers.”
Automatic, In-Domain, Question/Answer-Set Generation, IPCOM000245124D (Feb. 10, 2016) describes “a system for automatically generating a set of domain-specific question-answer (QA) pairs from a domain-specific corpus and an existing set of domain-general QA pairs.”